S260: Hetzner Multi Server Infrastructure And SandboxPlacementRouter

Documentation for Plans

S260: Hetzner Multi-Server Infrastructure and SandboxPlacementRouter

Session: 260 | Date: 2026-03-23 | Branch: session-254-259 (shared, concurrent) Agent: Claude Code (Opus 4.6) | Status: Phase 1 complete, Phase 2 ready to build Spec: docs/superpowers/specs/2026-03-23-hetzner-dual-server-infrastructure-design.md MCP Decisions: dev_1774275166101_7f7cvcj (infra), dev_1774280349838_q915b9u (architecture)

1. Session Mission

Set up GoGrab's second Hetzner dedicated server (AX102-U) and design a multi-server architecture where both servers work concurrently — not as primary/fallback, but as a unified fleet with role-based workload routing.

The ultimate goal: customer sandbox VMs route to the more powerful AX102-U, executive/dev VMs route to AX42-U, git operations run on AX102-U, and if either server hits capacity, work overflows to the other.

This requires a new SandboxPlacementRouter service that replaces the current single-server SelfHostedProvider in the sandbox provider chain.

2. Work Completed

Physical Infrastructure (fully done)

Configuration Changes (done, uncommitted)

New Files Created

Architecture Decisions

3. Current System State

What Works

What Partially Works

What Does NOT Work Yet

What Exists But Is Not Functional

4. Critical Gaps

Gap 1: SandboxPlacementRouter Does Not Exist

Why it matters: Without this, both servers cannot work concurrently for VM workloads. Only one server receives VMs at a time. Root cause: SelfHostedProvider is a single-URL HTTP client. SandboxProviderFactory selects ONE provider at startup. Impact: Executive VMs can't be routed to AX42-U. No overflow capacity. $248/mo of hardware with only half being used for VMs.

Gap 2: No Workload Type Signal in the Sandbox Creation Path

Why it matters: Even with a router, getOrCreateSandbox(sessionId, userId) has no parameter to indicate whether this is a customer VM or executive VM. Root cause: The SandboxProvider interface doesn't include workload type. ComputerUseGoal has persistentVm and user info but this isn't passed to the provider. Impact: Router can't make role-based decisions without this signal.

Gap 3: Placement Map for Sticky Routing

Why it matters: Once a VM is created on ax102-01, all subsequent operations (screenshot, click, destroy) MUST go to ax102-01. Without a mapping, operations would be sent to the wrong server. Root cause: SandboxHandle doesn't include serverId. The provider tracks sandboxes in an in-memory Map<sessionId, SandboxHandle> within a single SelfHostedProvider instance. Impact: Operations on existing VMs would fail with 404 if sent to the wrong server.

Gap 4: No Commits from This Session

Why it matters: All changes are uncommitted due to concurrent session constraint. If the working tree is reset, all config changes and new files are lost. What to do: The next session should commit all S260 changes before doing anything else.

5. Prioritized Recommendations

Tier 1 — High Leverage / Low Effort

T1.1: Commit all S260 changes

T1.2: Restart backend-ai-api to show both servers in dashboard

Tier 2 — Medium Effort / High Impact

T2.1: Build SandboxPlacementRouter

T2.2: Wire PlacementRouter into factory + pass workload type

Tier 3 — Large / Transformative Work

T3.1: IDS (Internal Dev Server) for executives

T3.2: Local AI inference on AX102-U

6. The One Highest-Leverage Next Step

Build SandboxPlacementRouter (T2.1 + T2.2).

This is the critical missing piece that turns $248/mo of hardware from "one active, one standby" into "both active with intelligent routing." Every other feature (executive VMs, overflow, monitoring, future servers) depends on the router existing. Without it, the second server is a $124/mo health-check endpoint.

7. Files / Systems to Inspect

Core Files for SandboxPlacementRouter Implementation

APIs/backend-ai-api/src/app/infrastructure/services/sandbox/
├── sandbox-provider.interface.ts    ← SandboxProvider interface (15 methods + 4 optional snapshot methods)
├── sandbox-provider.factory.ts      ← Factory that selects provider (modify to add PlacementRouter)
├── self-hosted.provider.ts          ← Current single-server HTTP client (extract client logic from this)
├── e2b-cloud.provider.ts            ← E2B provider (reference for interface implementation)
├── playwright.provider.ts           ← Playwright fallback (reference)
├── index.ts                         ← Barrel exports (add new exports)
└── [NEW FILES GO HERE]              ← sandbox-placement-router.ts, etc.

Consumer Files (should NOT need changes, but verify)

APIs/backend-ai-api/src/app/infrastructure/services/computer-use-loop.service.ts
  ← Lines 306-349: sandbox creation flow, needs workload type hint
  ← Lines 356-391: cleanup flow (snapshot or destroy)

APIs/backend-ai-api/src/app/infrastructure/controllers/sandbox-proxy.controller.ts
  ← Lines 1-280: proxies to SelfHostedProvider directly for file ops
  ← WARNING: This controller injects SelfHostedProvider directly, not via factory!
  ← Must be updated to use PlacementRouter or factory

APIs/backend-ai-api/src/app/infrastructure/controllers/observatory.controller.ts
  ← Lines 82-137: sandbox-providers endpoint, reads from factory

APIs/backend-ai-api/src/app/infrastructure/infrastructure.module.ts
  ← Lines 282-285: Current provider registrations
  ← Lines 347: SandboxProviderFactory export

Configuration Files

.env.local                          ← SANDBOX_SERVERS_JSON already defined
libs/libs-foundational/api-config/proxy.conf.js  ← Updated to ax102-01
deployment/sandbox-orchestrator/nginx.conf        ← Fixed /git/ route + body size
deployment/sandbox-orchestrator/deploy-orchestrator-all.sh  ← Fleet deploy

Critical Interface to Review

The SandboxProvider interface at sandbox-provider.interface.ts defines 15 required methods + 4 optional snapshot methods. The PlacementRouter must implement ALL of them. Key complexity:

IMPORTANT: SandboxProxyController Injects SelfHostedProvider Directly

sandbox-proxy.controller.ts has:

constructor(
  private readonly factory: SandboxProviderFactory,
  private readonly selfHostedProvider: SelfHostedProvider,  // ← DIRECT INJECTION
)

The writeFile, readFile, injectSshKey methods call selfHostedProvider.writeFile() directly (not via factory). This must be updated to route through the PlacementRouter for file operations to work on the correct server.

8. Validation Checklist

Before Making Changes

After Building PlacementRouter

Edge Cases to Test

9. MCP and Codebase Guidance

Start the next session with:

preflight({ task: "S261: Build SandboxPlacementRouter for multi-server VM routing", sessionNumber: 261 })
find_similar({ query: "sandbox provider factory multi-server routing placement" })
intelligence({ action: "recall", situation: "SandboxPlacementRouter architecture S260" })
convention({ action: "get", category: "infrastructure" })

After implementation:

log_decision({ type: "implementation", question: "SandboxPlacementRouter implementation", ... })
outcome({ action: "report", decisionId: "dev_1774280349838_q915b9u", wasSuccessful: true/false })

Key past decisions to recall:

10. Server Fleet Reference

ID Hardware IP DNS API Key Roles
ax102-01 AX102-U #2946196 (16C/32T, 128GB, 1.71TB) 178.63.165.83 ax102-01.gograb.space hNwl3fka8Jc_xuUFvdvWin2IPMfI38sL5XuSpKQD1MI customer-sandbox, git-operations
ax42-01 AX42-U #2938512 (8C/16T, 128GB, 1TB) 144.76.69.55 sandbox.gograb.space W8Wp0ltL8El_qRnXyoUoD0ffuSfZzmbn7Y7SVrx1k8s executive-sandbox, customer-sandbox (overflow)

SSH access:

COPY/PASTE PROMPT FOR SESSION 261

S261: Build SandboxPlacementRouter for multi-server VM routing.

CONTEXT:
Session 260 set up our second Hetzner server (AX102-U, ax102-01.gograb.space) and designed a
multi-server architecture. Both servers are running sandbox-orchestrator and healthy. But the
codebase still uses a SINGLE-SERVER SelfHostedProvider. We need a SandboxPlacementRouter that
routes workloads to the right server based on type, health, and capacity.

CURRENT STATE:
- ax102-01 (AX102-U, 32 CPUs, 60 VMs): RUNNING at https://ax102-01.gograb.space
- ax42-01 (AX42-U, 16 CPUs, 30 VMs): RUNNING at https://sandbox.gograb.space
- SANDBOX_SERVERS_JSON is defined in .env.local but NO code reads it yet
- SelfHostedProvider is single-URL, SandboxProviderFactory picks ONE provider
- SandboxProxyController injects SelfHostedProvider DIRECTLY (must be updated)
- All S260 config changes are UNCOMMITTED — commit first!

MISSION:
1. Commit all S260 changes: [S260.1] feat: Hetzner AX102-U setup + multi-server infrastructure config
2. Build SandboxPlacementRouter implementing SandboxProvider interface:
   - sandbox-server-client.ts — HTTP client per server (extracted from SelfHostedProvider)
   - sandbox-server-registry.ts — Parse SANDBOX_SERVERS_JSON
   - sandbox-health-monitor.ts — Per-server health polling + circuit breaker
   - sandbox-placement-map.ts — vmId→serverId sticky routing (in-memory, rebuild on restart)
   - sandbox-placement-router.ts — Filter→Score→Place, implements all 19 SandboxProvider methods
3. Wire into SandboxProviderFactory (priority 0 when SANDBOX_SERVERS_JSON set)
4. Add workloadType hint to getOrCreateSandbox() — pass from ComputerUseLoopService
5. Fix SandboxProxyController to route file ops through PlacementRouter
6. Test: VM creation on ax102-01, operations work, overflow to ax42-01

KEY FILES:
- APIs/backend-ai-api/src/app/infrastructure/services/sandbox/ (all files)
- APIs/backend-ai-api/src/app/infrastructure/services/computer-use-loop.service.ts
- APIs/backend-ai-api/src/app/infrastructure/controllers/sandbox-proxy.controller.ts
- APIs/backend-ai-api/src/app/infrastructure/infrastructure.module.ts
- .env.local (SANDBOX_SERVERS_JSON config)

ARCHITECTURE (from S260 research):
- Filter→Score placement (K8s/Nomad inspired, simplified for 2-3 servers)
- Roles: customer-sandbox, executive-sandbox, git-operations
- Priority: ax102-01 = priority 1 for customer, ax42-01 = priority 2 (overflow)
- Sticky routing: placement map tracks which server owns each VM
- Health: per-server circuit breaker (healthy → degraded → down)
- Backward compat: if SANDBOX_SERVERS_JSON not set, fall back to SANDBOX_SERVER_URL

MCP DECISIONS TO RECALL:
- dev_1774280349838_q915b9u — PlacementRouter architecture design
- dev_1774275166101_7f7cvcj — Dual-server role assignment

WARNINGS:
- SandboxProxyController injects SelfHostedProvider DIRECTLY for writeFile/readFile/injectSshKey
- SandboxHandle doesn't include serverId — PlacementRouter must track this in its placement map
- getUserVmState() and restoreFromSnapshot() must query the correct server per user
- getActiveSandboxes() must aggregate across ALL servers

SPEC: docs/superpowers/specs/2026-03-23-hetzner-dual-server-infrastructure-design.md
HANDOFF: docs/english/Plans/S260: Hetzner Multi-Server Infrastructure and SandboxPlacementRouter.md

Generated by Claude Code (Session 260.1) — March 23, 2026